Repository navigation
Disallow positional struct casting when field names don’t overlap - #19955
Merged
Merged
Conversation
Remove positional struct field mapping, enforcing name overlap for struct casts and document new behavior in casting logic and validation errors. Update struct-cast tests to assert new no-overlap error conditions, including equal/unequal field counts and cast-time rejection scenarios.
Contributor
|
I think we'll need an entry in |
kosiew
force-pushed
the
positional-fallback-19841
branch
from
January 26, 2026 10:56
d031ce0 to
fc32f78
Compare
kosiew
marked this pull request as ready for review
January 26, 2026 14:11
adriangb
approved these changes
Jan 26, 2026
adriangb
left a comment
Contributor
There was a problem hiding this comment.
Some relatively minor nits
| /// | ||
| /// This is useful for validating struct compatibility when casting between structs, | ||
| /// ensuring that source and target fields have overlapping names. | ||
| pub fn has_one_of_more_common_fields( |
Contributor
There was a problem hiding this comment.
Does this need to be pub?
Contributor
Author
There was a problem hiding this comment.
Yes, this function is re-used at
datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs#L662
// Skip const-folding when there is no field name overlap
if !has_one_of_more_common_fields(&source_fields, target_fields) {
return false;
}
| // Use struct builders or row constructors that preserve your mapping logic | ||
| ``` | ||
|
|
||
| **Why this change:** |
Contributor
There was a problem hiding this comment.
I think it's worth mentioning that this matches DuckDBs behavior
|
|
||
| ```sql | ||
| -- Explicitly rename to match source field names | ||
| SELECT CAST(source_col AS STRUCT<x INT, y INT>) FROM table1; |
Contributor
There was a problem hiding this comment.
Is this something other engines support? I would have guessed this fails under the "no name overlap" rule.
kosiew
marked this pull request as draft
January 27, 2026 02:42
kosiew
marked this pull request as ready for review
January 27, 2026 05:33
adriangb
approved these changes
Jan 27, 2026
Contributor
|
I plan to merge this tomorrow morning EST if there is no further feedback |
Contributor
|
Thanks for merging up @alamb ! |
Contributor
|
Thanks -- yeah I am just trying to keep the code flowing |
de-bgunter
pushed a commit
to de-bgunter/datafusion
that referenced
this pull request
Mar 24, 2026
…ache#19955) ## Which issue does this PR close? * Closes apache#19841. ## Rationale for this change Struct-to-struct casting previously fell back to **positional mapping** when there was **no field-name overlap** and the number of fields matched. That behavior is ambiguous and can silently produce incorrect results when source/target schemas have different field naming conventions or ordering. This PR makes struct casting **strictly name-based**: when there is no overlap in field names between the source and target structs, the cast is rejected with a clear planning error. This prevents accidental, hard-to-detect data corruption and forces callers to provide explicit field names (or align schemas) when casting. ## What changes are included in this PR? * Removed the positional fallback logic for struct casting in `cast_struct_column`; child fields are now resolved **only by name**. * Updated `validate_struct_compatibility` to **error out** when there is **no field name overlap**, instead of allowing positional compatibility checks. * Updated unit tests to reflect the new behavior (no-overlap casts now fail with an appropriate error). * Updated SQLLogicTest files to construct structs using **explicit field names** (e.g. `{id: 1}` / `{a: 1, b: 'x'}` or `struct(… AS field)`), avoiding reliance on positional behavior. * Improved error messaging to explicitly mention the lack of field name overlap. ## Are these changes tested? Yes. * Updated existing Rust unit tests in `nested_struct.rs` to assert the new failure mode and error message. * Updated SQLLogicTest coverage (`struct.slt`, `joins.slt`) to use named struct literals so tests continue to validate struct behavior without positional casting. ## Are there any user-facing changes? Yes — behavioral change / potential breaking change. * Casting between two `STRUCT`s with **no overlapping field names** now fails (previously it could succeed via positional mapping if field counts matched). * Users must provide explicit field names (e.g. `{a: 1, b: 'x'}` or `struct(expr AS a, expr AS b)`) or ensure schemas share field names. * Error messages are more explicit: casts are rejected when there is “no field name overlap”. ## LLM-generated code disclosure This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed and tested. --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
This was referenced Nov 8, 2026
zzcclp
pushed a commit
to zzcclp/arrow-datafusion
that referenced
this pull request
Nov 10, 2026
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes apache#14396 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. Please explain the problem you are trying to solve in terms of the user-visible behavior, rather than the implementation. For example, "The code in `foo.rs` doesn't handle nulls" is a symptom of the implementation. "COUNT(DISTINCT) returns wrong results when the column contains nulls" is the user-visible problem. --> Type coercion for structs could result in incorrect results. Namely, because of a fallback to coercing by position structs with different fields could end up returning results when an error is the expected result. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here, but it is sometimes worth providing a summary of the individual changes in this PR. --> We remove the position fallback which follows a previously established pattern [here](apache#19955) for struct casting. We also ensure struct field names match before moving to struct coercion. Commit [c4813bf](apache@c4813bf) -> Baseline sqllogictests that show the expected failure and do not pass without fix Commit [c3afa0d](apache@c3afa0d) -> Remove position based fallback and require struct field names to match ## What is the testing strategy for this PR? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code Briefly describe how this PR is tested, and point to the specific tests you added. For example: 'This new feature is covered by the `sqllogictest` cases added in `foo.slt`'. If this PR does not add tests, explain why. For example, if the change is already covered by existing tests, please mention it. You should also check the `codecov` bot reply on this PR to confirm the changed code is exercised. --> A initial commit with tests demonstrating the expected behavior were added and then fixes applied on top of that. The tests were focused on coercion of struct types when using `UNION`, `CASE`, and comparison between struct types. ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please add the `api change` label. --> Errors that previously incorrectly succeeded will now fail and this class of error is handled completely during type coercion now so errors will be reflected as such.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
Struct-to-struct casting previously fell back to positional mapping when there was no field-name overlap and the number of fields matched. That behavior is ambiguous and can silently produce incorrect results when source/target schemas have different field naming conventions or ordering.
This PR makes struct casting strictly name-based: when there is no overlap in field names between the source and target structs, the cast is rejected with a clear planning error. This prevents accidental, hard-to-detect data corruption and forces callers to provide explicit field names (or align schemas) when casting.
What changes are included in this PR?
cast_struct_column; child fields are now resolved only by name.validate_struct_compatibilityto error out when there is no field name overlap, instead of allowing positional compatibility checks.{id: 1}/{a: 1, b: 'x'}orstruct(… AS field)), avoiding reliance on positional behavior.Are these changes tested?
Yes.
nested_struct.rsto assert the new failure mode and error message.struct.slt,joins.slt) to use named struct literals so tests continue to validate struct behavior without positional casting.Are there any user-facing changes?
Yes — behavioral change / potential breaking change.
STRUCTs with no overlapping field names now fails (previously it could succeed via positional mapping if field counts matched).{a: 1, b: 'x'}orstruct(expr AS a, expr AS b)) or ensure schemas share field names.LLM-generated code disclosure
This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed and tested.